The present disclosure relates generally to compressing systems management data without excessive loss of original detail, more particularly but not by way of limitation, to a method and system for dynamically selecting compression settings according to the data being collected.
To improve storage or transmission efficiency, data compression stores data in a format that requires less space than required to store the original raw data. Data compression can be lossless or lossy. Lossless compression of time series data can be impractical due to the typically high overhead. For example, using run length encoding or dictionary-based compression may require a dictionary or large buffer to be maintained and may require great care to avoid data loss. Lossy data compression does not store all of the data and instead discards data deemed irrelevant, thereby reducing the overall amount of data stored. However, potentially valuable information may be lost in the process.
Although different forms of data compression have been used in chemical process control or the like, using data compression for systems management data poses special challenges due to the very dynamic nature of the data. In FIG. 1, an enterprise computing environment 100 has many different types of computer and network components. Each of these diverse components requires monitoring and collection of time series data for historical analysis to aid in problem diagnosis or other system management functions such as capacity planning. For example, CPU utilization on one system 110 is very unlikely to behave in the same way as CPU utilization on another system 120. Similarly, disk utilization measurements may vary widely across the different disk units 130 in the enterprise computing environment 100. Moreover, each component of the environment 100 may have to be measured using a different scale. For example, CPU utilization may be measured in the range 1-100 and reflect a percentage of CPU usage, whereas a disk unit may be measured in a virtually unlimited range reflecting the amount of free space available.
As expected, systems management produces large quantities and diverse forms of time series data that may need to be retained for historical reference. Storing all of the raw data is potentially unfeasible. Therefore, some form of data compression may be necessary to limit the storage space required or to reduce the amount of I/O required to store the data. Traditionally, compression of systems management data typically uses geometric averaging to reduce the granularity of the stored data over time. While this type of compression is effective at reducing the volume of data stored, it loses much of the detail of the original data, which is especially true when time series data is filtered according to a compression ratio.
Defining appropriate compression deadbands for systems management data is difficult due to the diverse systems management data collected in the environment 100. For example, a chemical process can have a maximum deviation defined for every measurement because hardware sensors operate within precisely defined tolerances to collect measurements and the collected measurements are largely invariant due to steady state operation. In systems management, establishing a maximum deviation acceptable for every metric is impractical because no “specific sensors” exist.
In systems management, components of the environment 100 and their metrics may also be discovered dynamically. For example, when a hard drive is added to the system, new metric data will be made available related to that hard drive. In addition, metrics may be measured according to various scales, ratios, measurements, etc. For example, a metric indicative of the amount of free memory can be measured in either megabytes or gigabytes, and specifying a tolerance band of ±1000 bytes can be very unacceptable for a small machine but acceptable for very large servers. Moreover, requiring a user to manually configure an appropriate deadband for each metric measured on each component of the environment 100 is impractical, especially when the enterprise computing environment 100 is diverse, complex, or dynamic.